OBJECT TAKING OUT APPARATUS 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to an apparatus for successively taking out 
a plurality of objects stacked randomly using an industrial robot. 

2. Description of Related Art 

In taking out randomly staked objects such as workpieces one by one 
by a robot by detecting the objects using a visual sensor, there is a difficulty in 
how to treat objects overlapped with each other. This problem is generally 
solved by finding out an object located at the uppermost position in the stack of 
objects and taking out the uppermost workpiece. 

In a method of detecting an interference of articles as disclosed in JP 
9-91441 A, it is determined whether pr not there is any article other than a 
detected article in a circumferential region thereof using an image of articles 
captured by a TV camera, and overlapping of the detected workpiece and the 
other workpiece is determined based on an areas of the image of the other 
workpiece in the circumferential region and a height of the other workpiece 
determined by the other method. In this interference detection method, if the 
object of detection has a planner simple shape, for example, the detection of 
the other object will be vague to lower reliability of detection of the 
interference of the articles. Further, since the method other than the image 
processing is required for obtaining the height of the other article, this method 
is disadvantageous in manufacturing cost. 

In an apparatus for recognizing position of an object as disclosed in JP 
8-27 1223 A, an object is detected by tracing an outline of an image of the 
object and if it is determined that there is overlapping of the object with other 
object, upper/lower positional relation of the detected object and the other 
object is determined based on a distribution of gradation of the overlapped 
portion of the image of the objects, or a height of the other object determined 
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by emission of a reference light. In this apparatus, the determination of the 
overlapping of the objects depends on the tracing of the outline of the image of 
the objects, and it is highly probable that the outline of the image of the objects 
is not continuous because of reflecting condition of illuminating light on the 
objects, to lower rehabiUty of the determination of the overlapped portion. 
Also, rehabiUty of determiuatiori pf the upper/lower position of the objects 
based on the distribution of gradation would be low for the same reason, and 
the emission of the reference light is disadvantageous in manufacturing cost of 
the apparatus. 

Further, in an overlapping state recognizing method as disclosed in JP 
9-539 15 A, an overlapped portion of detected objects is found out based on 
relative positions of the detected objects and upper/lower position of the 
objects is determined by judging what object the overlapped portion belongs to 
by local matching of features of the objects in the overlapped portion. In this 
method, the overlapped portion of the objects is determined by considering 
correlation of shapes of the objects geometrically. The processing for the 
detennination would be simple in the case where each object has a simple 
shape such as a circle having a circumference at the same distance from a 
center of the shape, but if the objects have complicated shapes, processing for 
the detennination will be compKcated to increase calculation cost for carrying 
out the complicated processing. 



SUMMARY OF THE INVENTION 



An object of the present invention is to provide an apparatus for 
successively taking out a plurality of randomly stacked objects according to 
priority pf taking out the object with high rehability and low cost. 

According to the present invention, position/orientation of images of 
the respective objects are determined in an image of the plurality of objects, 
and visuality of partial features of the object is evaluated by the number of 
successes in detection of the partial features, the sum of matching indexes each 
representing a degree of matching of a partial image of the object and the 
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corresponding partial feature, or the sum of products of the matching indexes 
and significance of the respective partial features with respect to a whole 
feature of the object, and the priority of taking out the object is determined 
based on the visuahty of the partial features of the objects. 

An object taking out apparatus of the present invention comprises: 
image capturing means for capturing an image of the plurality of objects; 
storing means for storing information on a whole feature representing a whole 
shape of the object, information on partial features representing one or more 
partial shapes of the object, and information on partial feature detecting regions 
defined for detection of the partial features of the object; object detecting 
means for determining positions/orientations of images of the respective 
objects in the image of the plurality of objects using the stored information on 
the whole feature of the object; partial feature detecting region setting means 
for setting partial feature detecting regions for the images of the respective 
objects based on the determined positions/orientations of the respective objects 
and the stored information on the partial feature detecting regions; partial 
feature detecting means for detecting the partial features of the objects in the 
set respective partial feature detecting regions; priority determining means for 
determining priority of taking out the object based on results of detection of the 
partial features of the objects; and a robot for successively taking out the 
objects according to the determined priority. 

The priority determining means may evaluate the result of detection of 
the partial features of the object by the number of successes in detecting the 
partial features and determine the priority of taking out the object based on the 
number of successes. 

The priority determining means may evaluate tiie result of detection of 
the partial features of the object using a matching index representing a degree 
of matching of a partial image of the object and the corresponding partial 
feature in the partial feature detecting region, and determine the priority of 
taking out the object based on the sxmi of the machining indexes. 

Further, the priority determining means may evaluate the result of 
detection of the partial features using the matching index and significance of 
the partial feature detecting region with respect to the whole feature of the 
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object, and determines the priority of taking out the object based on the sum of 
products of the matching iadexes and the significance of the partial features. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a schematic block diagram showing a system configuration of 
an object taking out apparatus including an image processing device for 
determining priority of taking out the object according to an embodiment of the 
present invention; 

FIG. 2a is diagram showing an example of whole feature information of 
one workpiece as an object of taking out and FIG. 2b is a diagram showing an 
example of partial feature information of the wor]q)iece; 

FIG. 3 is a diagram showing an example of partial feature detection 
regions in the whole image of the workpiece; and 

FIG. 4a is a diagram showing images of two workpieces overlappiag 
with each other, and FIG. 4b is a diagram showing partial feature detecting 
regions respectively set to the images of the two workpieces for determining 
priority of taking out the workpiece. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

A system configuration of an object taking out apparatus and 
processing to be performed by the object taking out apparatus according to an 
embodiment of the present invention will be described referring tb FIGS . l-4b. 

As shown in FIG. 1, a plurality of workpieces 41 is stacked randomly 
in a workpiece container 40. The workpiece taking out apparatus comprises a 
robot 3 1, a robot controller 30 for controlling the robot 31, and a 
two-dimensional sensor including a video camera 1 and an image processing 
device 20. The image processing device 20 indicated by the broken line in 
FIG. 1 is shown by a functional block diagram representing functions which 
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are achieved by hardware such as a processor, memories, interfaces and a 
display device, and software such as an operation program and appUcation 
programs for image processing. 

The robot 3 1 is arranged at a position suitable for successively taking 
out the workpieces 41 placed on the workpiece container 40, and the video 
camera l is arranged at a position where a whole image of tiie workpieces 41 
on the workpiece container 40 is captured within a field of view of the camera 
1 fi*om approximately exactly above the workpieces 41 . The robot controller 
30 and the image processing device 20 are connected with each other by a 
communication line 32, and the robot controller 30 controls the root 31 to 
successively perform handling, i.e. approaching, holding and taldng out of the 
workpieces 41 according to "priority" of handling of the workpieces 41 
determined by analyzing the image captured by the video camera 1 by the 
image processing device 20. 

The "priority" of handling is a degree representing appropriateness or 
easiness of taking out the workpiece next. The taking-out operation of the 
workpieces 41 according to this embodiment is carried out by repeatedly 
performing the following steps (i) and (ii) imtil the taking out of all of the 
workpieces 41 is completed. 

(i) determining priority of handling in the randomly stacked workpieces 
41 by analyzing status of the respective workpieces 41 by the visual sensor, 
and 

(ii) controUiag the robot 31 to perform handling (approaching, holding 
and taking out) of the workpiece of the first priority of handling using data of 
position/orientation of the workpiece of the first priority. 

The handling priority of the last workpiece left on the workpiece 
container 40 is automatically determined to be the first. 

For the determination of priority of handling in the above step (i), 
whole feature information to constitute a model of one of the workpieces as 
objects of taking out, and partial feature information to constitute partial 
models of the workpiece are prepared in advance. 

An image of one of the workpieces 41 is captured by the video camera 
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1 arranged at a reference position. A model creating section 6 of the image 
processing device extracts whole feature information 8 of the workpiece 41 
from the captured image of the workpiece 41 . The whole feature information 
8 is stored in an image memory 1 1 of the image processing device 20. The 
whole feature information 8 represents a feature of the whole shape of the 
workpiece 41 to constitute a model of the A example of the 

whole feature information 8 is shown in FIG. 2a. 

Further, partial feature information 10 of the workpiece 41 is extracted 
from the captured image of the workpiece 41 by a partial model creating section 
7 and stored in the image memory 1 1 with partial feature detecting region 
information 9. 

The partial feature information 10 represents features of shapes of one 
or more parts of the workpiece 41 to constitute partial models of the workpiece. 
FIG. 2b shows an example of the partial feature information 10 to be associated 
with the whole feature information 8 which is extracted form the same image 
of the workpiece 41 as shown in FIG. 2a. In FIGS. 2a and 2b, an image 
coordinate system in a screen of tiie display device is indicated by O-XY. 

As can be see from comparison of FIG. 2a and FIG. 2b, the whole 
feature infomiation 8 shovm in FIG. 2a represents the whole shape of the 
workpiece 41 rather than any particular partial shape of the workpiece 41 . In 
this example, the whole feature information 8 comprises principal contours of 
the whole image of the workpiece 41 as viewed from above the worlq)iece. 
The partial feature information 10 shown in FIG. 2b comprises contours of six 
parts selected in the principal contours of the whole image of the workpiece 41 
constituting the whole feature information 8. The respective regions in which 
the six partial features exist constitute partial feature detecting regions 51-56, 
and information thereof is stored in the image memory 1 1 as partial feature 
detecting region information 9. In other words, the contours of the workpiece 
41 in the respective partial feature detection regions 51-56 constitute the partial 
feature information 10 in this example. A relative position of the partial 
feature detecting regions 51-56 with respect to a reference point 50 of the 
whole feature information 8 is included in the partial feature detecting region 
information 9. 
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The processing for determination of the priority of handling of the 
workpiece for successively handling the workpieces will be described. An 
image of the plurality of workpieces 41 captured by the video camera 1 is sent 
to a whole feature detecting section 2 of the image processing device 20. The 
whole feature detecting section 2 analyzes the captured image of the 
workpieces 41 to successively riecognize whole images of the respective 
workpieces 41 using the whole feature information 8 of the workpiece 41 and 
determine positions/orientations of the respective workpieces 41 . The 
recognition of images of the workpieces in the captured image can be carried 
out by various known technologies. A partial feature detecting region setting 
section 3 sets partial feature detecting regions in which the partial features are 
to be detected based on the information on positions/orientations of the 
respective workpieces 41 determined by the whole feature detectiug section 2 
and the partial feature detecting region information 9 stored in the image 
memory 11. 

An example of partial feature detection regions in the whole image of 
the workpiece 41 is shown in FIG. 3. As shown in FIG. 3, a whole image of 
the workpiece 41 is recognized in the captured image of the workpieces 41 
using the whole feature information 8, and the position/orientation of the 
workpiece 41 are determined by the whole feature detecting section 2. In this 
example, the whole image of the workpiece 41 is inclined in the image 
coordinate system O-XY, and position of the reference point of the whole 
image of the workpiece 41 is determined as indicated by a reference numeral 
60. Partial feature detection regions 61-66 are set by the partial feature 
detecting region setting section 3 based on the partial feature detecting region 
information 9 using the determined reference point 60 and the orientation of 
the whole image of the workpiece 41 . Then, the detection of the partial 
features in the respective partial feature detection regions 61-66 is performed 
by the partial feature detecting section 4 using the partial feature information 
10. 

A result of detection of the respective partial features is expressed by a 
positive integer representing the sum of successes in detection of partial 
features, for example, and the positive integer is calculated for each workpiece. 
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Data of the detection result of the partial features as evaluated are sent to a 
handling priority determining section 5, and handling priorities are assigned to 
the workpieces 41 in the order of the positive integer representing the number 
of successes in detection of the partial features. Thus, flie first priority is 
assigned to the workpiece having the greatest positive integer and the stored 
information on positioii/orientation of the w6rlq}iece of the first priority is sent 
to the robot controller 30. If there are a pluraHty of workpieces having the 
same and greatest positive integer, the first priority is assigned to the 
workpiece detected earher, for example. 

An example of the determination of handling priorities will be 
described on a case where whole images of two workpieces A and B are 
positioned as shown in FIG. 4a. The partial features detecting regions as 
indicated by the broken lines in FIG. 4b are set to the respective whole images 
of the two workpieces and the detection of the partial features in the respective 
partial features detecting regions is performed, as described. The number of 
successes in detection of the partial features in the lefthand workpiece B is 
smaller than that in the right hand workpiece A. Thus, the first priority is 
assigned to the workpiece A as an object to be picking up first. 

As the measure or criterion for determining priorities of picking up, a 
matching index representing a degree of matching of the partial image of the 
workpiece and the corresponding partial feature information may be used, and 
the priorities may be determined based on the sum of the matching indexes, as 
well as the number of success in detection of the partial features as described. 
The matching index may be expressed by a positive integer ranging fi*om the 
minimum value of "0" to the maximum value of *'100", for example, and is 
determined for each detection of the partial feature in the partial feature 
detecting region of the workpiece. 

For example, with respect to the workpiece A as shown in FIGS. 4a and 
4b, since the whole shape of the workpiece A is exposed to be visible, the 
maximum value of "100" of the matching index is given for each detection of 
the partial feature to make the sum of "600" of the matching indexes. With 
respect to the workpiece B, since a part of the workpiece B is covered by the 
workpiece A, a lower value, e.g. "30" of the matching index is given to the 
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detection of the partial feature in the covered portion of the workpiece, to make 
the sum lower than the simi "600" of the matching indexes of the workpiece A. 
With this arrangement, it is determined that the priority of the workpiece A > 
the priority of the workpiece B more minutely. 

Further, so as to take account of significance of the respective partial 
features, it is effective to use data of areas of the partial feature detecting 
regions, and the priorities may be determined based on the sum of products of 
the respective matching indexes and the areas of the corresponding partial 
feature detecting regions. Specifically, the matching index in each detection 
of the partial feature is multiplied by a coefficient representing significance of 
the partial feature detecting region with respect to the whole feature, and the 
priority of each detected workpiece is determined based on the sum of the 
products. 

The above embodiment is directed to a case where the robot handles 
two-dimensional planner workpieces using the single video camera. However, 
the present invention is not restricted to such case, and is applicable to a case 
where a plurality of three-dimensional solid worlq)ieces are to be handled by 
the robot using a stereo vision system with a plurality of video cameras. In 
this case, the present invention is applied to an image of workpieces captured 
by any of the video cameras, so that the priority for successively picking up the 
pluraUty of workpieces is determined. 

According to the present invention, the priority of picking up of the 
object is determined b£ised on visuaUty of partial features of the objects, to 
solve the problems in the prior art as follows; 

(1) The determination of the priority is not concerned with reliability of 
determination of the other object which has been a problem in determination of 
overlapping of the objects in the prior art. 

(2) The determination of the priority does not concem continuousness 
of an outiine of an image of objects which has been a problem in determination 
of overlapping of the objects in the prior art. 

(3) The evaluation of distribution of gradation of the image of the 
objects for determinating upper/lower relation of the objects in the overlapped 
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portion, which is of low reliability, is not necessary and also the emission of a 
reference light is not necessary. 

(4) It is not necessary to carry out complicated processing for the 
complicated shapes of the objects in order to determine the overlapped portion 
with other object, which has been required in the prior art. 

Thus, the present invention provides an object taking out apparatus 
advantageous in reliabiUty of operation and also in cost for processing for 
operation. 
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